Let’s take what we’ve learned in one dimension and apply it to a two dimensional image.
Remember how edges are related to gradients; we developed some operators that you can apply to gradients and we discussed first applying filters to smooth out the noise and in what order that can be done to save operations.
In 2D it’s not just a derivative but also the direction of the derivative.
Suppose we want to take the derivative in the x direction.
\[(I\otimes g) \otimes h_x = I \otimes (g \otimes h_x)\]
What this sub x is implying is that we are only applying the derivative in the x direction. This could be the Sobel operator or any other operator that requires a derivative. It’s a small filter taking a derivative. g is the gaussian function and the associative property allows for moving the operation around.
This gives us the derivative of the gaussian smoothed image from before. It’s preferable to do it this way, but why?
Why is it preferable to apply \(h\) to the smoothing function \(g\) and apply the result to the image?
Is the horse dead yet? The smoothing function \(g\) is generally smaller than the image. Applying \(h\) to the smaller function requires fewer operations that then don’t need to be repeated.
Smoothed Gaussian
How big of a gaussian should we use? What effect does a changing spread have on the derivative? As your \(\sigma\) grows your smoothing effect will magnify; the same effect holds true when computing derivatives, enhancing the magnitude of the derivative as it grows in space.
What effect does this have in practice? Smaller values will capture finer features as edges and larger values will filter out to only large features.
How do we actually find the edges?
This operation, the Canny operator, works as follows:
In python this is called with
import cv2
im = cv2.imread('image')
cv2.Canny(im,...)
So, let’s walk through Canny edge detection in practice.
Given some image, we would like to detect edges using the method that Canny created.
Although this looks like edges to a person, it’s simply the magnitude of the gradient. It needs to be worked on prior to being useful.
Getting rid of anything in the image that is above / below a threshold allows us to quickly and easily discard things that most likely are not edges. Lena got a haircut…
Thinning allows representing large and wide bands of ‘edge’ as thin and simple contour lines by checking for local maxima along the gradient direction.
Along the direction of the gradient, as seen in the images to the right, there will be a local maxima somewhere on the line of the gradient direction. The ‘thinned’ edge is represented solely by that local maxima. This can require subpixel accuracy when interpolating!
This assumes that noisy edges that are close to strong edges are likely edges.
This code block reads in the frizzy and froomer images, uses a Canny filter to compute the edges in the image, and then uses a boolean operation to detect where edges in the two images overlap.
import cv2
# Helper function to show images.
from cs6476helpers import imshow
# Read in the data
frizzy = cv2.imread("frizzy.png")
froomer = cv2.imread("froomer.png")
# Show off the original images
imshow(frizzy)
imshow(froomer)
# Calculate the edges using the Canny filter
# Note that 10 and 20 for thresholds here are arbitrary.
frizzy_edges = cv2.Canny(frizzy, 10, 20)
froomer_edges = cv2.Canny(froomer, 10, 20)
# Show off the edges
imshow(frizzy_edges)
imshow(froomer_edges)
# Overlay the edges
shared_edges = frizzy_edges & froomer_edges
# And show off where they overlap!
imshow(shared_edges)
What is the secret code in the image above? (cue James Bond music…)
It’s really hard to know when an edge image is good. What will you use them for?
The Canny operator tends to be better at pulling out edges you want to use for future processing.
What size of kernel should we use in a Canny filter?
Large \(\sigma\) detects large edges and small detects small.
Is the Canny operator sensitive to noise? This is mostly false and is dependent on the \(\sigma\) chosen for the smoothing operation. The smaller the \(\sigma\) the more likely the image will be to pick up noise.
The 1D case of the second derivative of the Gaussian was an ‘inverted Mexican hat’ operator. The ‘zero crossings’ of the second derivative correspond to the edges.
There’s a problem! How does this extend to two dimensions? If you’re calculating second partial derivatives then you have \(f_{xx}\), \(f_{xy}\), \(f_{yy}\), and \(f_{yx}\). Which do you use?
Some of them; Use the Laplacian.
\[ \nabla ^ 2 h = \frac{ \partial ^ 2 f }{ \partial x ^ 2 } + \frac{ \partial ^ 2 f }{ \partial y ^ 2 } \]
This is a single operator that can find edges on both the X and the Y. Yay! Do the Mexican sombrero dance.
import cv2
# Helper functions
from cs6476helpers import imshow
from cs6476helpers import surf
from cs6476helpers import norm
from cs6476helpers import LoG
# Load in the image and display it
img = cv2.imread("lena.png")
imshow(img,title="Lena")
# Make the image into a grayscale image and display it
img2 = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
imshow(img2,title="Boring Lena")
# Make a Gaussian filter and plot as surface
filter_size = 10
filter_sigma = 3
gaussian = cv2.getGaussianKernel(filter_size,filter_sigma)
gaussian = gaussian * gaussian.T
surf(gaussian)
# Apply the filter and show the smoothed image
lena_smoothed = cv2.filter2D(img2,-1,gaussian)
imshow(lena_smoothed,title="Blurry Lena")
# Method one, shift differential. Shift the image to L, then R, then show the difference.
lena_left = np.roll(lena_smoothed,-4,1)
lena_right = np.roll(lena_smoothed,4,1)
imshow(norm(lena_left*1.+lena_right*1.))
# Method two, Canny filter on both the unsmoothed and smoothed images
canny_lena = cv2.Canny(img2, 60, 110)
imshow(canny_lena)
uncanny_lena = cv2.Canny(lena_smoothed, 60, 110)
imshow(uncanny_lena)
# Method three, Laplacian of Gaussian
laplacian = LoG(4, 1.)
surf(laplacian)
laplacian_edges = cv2.filter2D(img2,-1,laplacian)
imshow(norm(laplacian_edges),title="Laplacian Lena")
Original Lena
Monochrome Lena
Gaussian Surface
Fuzzy Lena
Shifted Lena
Laplacian of Gaussian
Canny Lena
Laplacian Lena
Uncanny Lena